Intel® Dynamic Load Balancer Driver for VMware ESXi* with Enhanced DirectPath I/O
=================================================================================

The Intel® Dynamic Load Balancer (Intel® DLB) is a PCIe device that provides
load balanced, prioritized scheduling of events across CPU cores enabling
efficient core-to-core communication. It is a hardware accelerator located
inside the latest Intel® Xeon® devices offered by Intel.

This software enables Intel® Scalable I/O Virtualization for the Intel® DLB
on VMware ESXi* hypervisor with Enhanced DirectPath I/O. Combination of Intel
and VMware technologies enables the creation of Virtual Devices (VDEVs) from
a single Intel DLB Physical Device to support hardware acceleration for
multiple guest Virtual Machines (VMs).

This document describes how to install Intel® DLB Driver on VMware ESXi*
hypervisor and how to make Intel® DLB HW available inside VMs via DLB VDEVs.
For more details on how to use Intel® DLB from a VM, please refer to the
documents mentioned in the "Documentation" section.

Note that this document assumes that the reader is familiar with virtualization
technologies and has some level of familiarity with the VMware ESXi* hypervisor.
This document does not explain how to install VMware ESXi*, how to create a VM
and how to administrate hypervisor using vSphere* Host Client or VMware vCenter*.
For more details, please refer to VMware documentation.

*Other names and brands may be claimed as the property of others.


License
=======
Refer to LICENSE.txt in this package for license information before using this SW.


Details/Limitations of this Release
===================================
 * This software release is intended for platforms that contain
      4th Generation Intel® Xeon® Scalable Processor (2710 DLB)
      5th Generation Intel® Xeon® Scalable Processor (2710 DLB)
 * This driver is only compatible with a particular ESXi version (VMware limitation)
 * This driver is mutually exclusive with DirectPath I/O drivers
 * VM re-configuration is required switching from DirectPath I/O drivers
 * Number of MSI-X interrupts per VDEV in a VM is limited to 128
 * This driver release doesn't support VMware vSphere* vMotion
 * Number of PCI passthrough devices per VM is limited
      https://kb.vmware.com/s/article/1003497


Documentation
=============
Associated Linux SW and collaterals can be found on the Intel® Download Center
    https://www.intel.com/content/www/us/en/download/686372


Install Intel® DLB VMware Driver
================================
1. Open a SSH connection to the target ESXi host (SSH needs to be enabled).

2. Copy the driver package to the ESXi
   (assume here and further that the target location is in '/tmp' folder and
   "scp" utility is used to copy the file to the target ESXi at 10.10.10.10)
    > scp dlb_*_rel_bin_*-dvx.tar.gz root@10.10.10.10:/tmp

3. Extract the driver package
    > cd /tmp
    > tar -xzf dlb_*_rel_bin_*-dvx.tar.gz

4. Install the driver from component
    > esxcli software component apply -d /tmp/*Intel-dlb*.zip

5. Reboot the system to complete driver installation
    > reboot

5. Reset module parameters to default values (if driver was installed early)
    > esxcli system module parameters clear -m dlb

6. Configure module parameters if needed. Please refer to these sections
    * Configure module parameters
    * DLB resource provision per VDEV

7. Verify the driver is loaded after reboot
    > esxcfg-module -l dlb | grep dlb
      dlb                      8    184
    > lspci -p | grep 2710
      0000:eb:00.0 8086:2710 8086:0000                V dlb
      0000:f0:00.0 8086:2710 8086:0000                V dlb
      0000:f5:00.0 8086:2710 8086:0000                V dlb
      0000:fa:00.0 8086:2710 8086:0000                V dlb


Configure module parameters
===========================
1. To list module parameters along with their descriptions
    > esxcli system module parameters list -m dlb

2. To restore module parameters to default values
    > esxcli system module parameters clear -m dlb

3. To set one module parameters
    > esxcli system module parameters set -m dlb -p name=value

4. To append a module parameter
    > esxcli system module parameters set -m dlb -a -p name=value

5. Driver needs to be reloaded to apply changes
    > esxcfg-module -u dlb
    > kill -HUP $(cat /var/run/vmware/vmkdevmgr.pid)

Note: If driver is used together with Intel® Accelerator Management Daemon for
VMware ESXi*, driver unload may report that "module symbols in use".  In this
case please proceed with system reboot or refer to daemon documentation on how
to temporally stop it.


DLB resource provision per VDEV
===============================
Amount of resources provisioned to VDEV is configured via module parameters
    > esxcli system module parameters set -m dlb -a -p provision_scale_factor=value

The "value" valid range is [0 - 16]. Default: 1. Recommended: 1, 2, 4.
0: No DLB provision per VDEV by default. Or custom specified by DLB tool/plugin.
M: DLB provision per VDEV is 1/M of all DLB resources, M is from [1, 16] range.


Assign VDEVs to VM via vCenter Client (recommended)
===================================================
1. Login to the vCenter Client.

2. Find target VM in the inventory. Ensure that the VM is powered off.

3. In the center pane, click "Edit Settings..." icon to edit VM configuration.
   A pop-up window with the VM settings will appear.

4. Click on the "ADD NEW DEVICE" menu and choose "PCI Device". A "Device
   Selection" pop-up will appear. In this new windows traverse to the end of
   list and choose "DLB Virtual Device" under "Enhanced DirectPath I/O" access type.

   Click "SELECT" button to confirm choice. Find new PCI device and specify a
   particular VDEV provision under "Provisioned resources" or leave a default one.
   Additional VDEVs could be added by repeating this step.

5. All memory for the VM must be reserved. Expand "Memory" and check that
   "Reservation" section is reporting "All VM memory is reserved for this VM.".
   Otherwise set "Reserve all guest memory (All locked)" checkbox.

6. Click "OK" to save VM configuration.


Assign VDEVs to VM manually via VM vmx file (not recommended)
=============================================================
1. Login to the vSphere Host Client on the target ESXi host.

2. In the left pane, expand "Virtual Machines" and select desired VM. Ensure
   that the VM is powered off.

3. Click on the "Edit" button to edit VM settings. A pop-up window will appear.

4. All memory in the VM must be reserved. Expand "Memory" and set
   "Reserve all guest memory" checkbox.

5. Open a SSH connection to the target ESXi host.

6. Locate VM Config file (usually at "/vmfs/volumes/<datastore name>/<vm name>")
   mentioned in VM settings ("VM Options" tab, "VM Config File" field inside
   expanded "General Options") and open it via "vi" editor.

7. Add the following VDEV parameters to assign VDEV0 to VM
       pciPassthru0.present = "TRUE"
       pciPassthru0.virtualDev = "dvx"
       pciPassthru0.dvx.deviceClass = "com.intel.dlb"
       pciPassthru0.dvx.config.profile_id = "-1"

   Additional VDEVs can be added by repeating this step with increasing number
   at `pciPassthru*` parameter group (`pciPassthru1`, `pciPassthru2` and so on).

   Supported "profile_id":
       "0":     Custom  (VDEV provisioned with custom amount of DLB resources)
       "-1":    Any     (VDEV provisioned with any amount of DLB resources)
       "1":     All     (VDEV provisioned with all DLB resources)
       "2":     1/2     (VDEV provisioned with 1/2 of DLB resources)
       "4":     1/4     (VDEV provisioned with 1/4 of DLB resources)

    By default all DLB devices provisioned with "profile_id" are set to "1".
    If "profile_id" is set to "-1" it means any available VDEV to be used for VM.
    The recommended way to update DLB device provision (load "profile_id") is to
    use a special vCenter* Plugin or Daemon configuration tool. Same also could
    be done manually via module params. In this case "profile_id" is set to
    "provision_scale_factor" if the latter one equals to "1", "2" or "4" and
    "snX_group_mode" are set to defaults. Otherwise "profile_id" is set to "0".


8. Save edited VMX file.

Notes:
 * vSphere Host Client doesn't support VDEVs assignment to VM via VM settings.
 * vSphere Host Client doesn't allow to change VM settings (VMware limitation)
   after VDEVs were added into VMX file. To do changes via UI comment out all
   VDEV-related settings in VMX file using "#" symbol, do changes via UI and
   then restore previously commented strings.


Uninstall Intel® DLB Driver
===========================
1. Open a SSH connection to the target ESXi host.

2. Reset module parameters to default values (not required but recommended)
    > esxcli system module parameters clear -m dlb

3. Unload module (not required and may fail if daemon is used)
    > esxcfg-module -u dlb

4. Uninstall driver
    > esxcli software component remove -n Intel-dlb

5. Reboot the system to complete driver uninstallation
    > reboot

Note: It's not mandatory to reset module parameters but ESXi keeps this
      information after driver is removed from the system.
